GRIM-U: A GRIM-like observation to establish impossible p values from ranked tests

The Medical Evidence Project uses forensic metascientific methods to examine medical research publications that have a disproportionately high impact on human health. Our aim is to determine where problematic data or claims have entered important areas of the medical literature with the ultimate goal of making medical care safer for patients.

Our work includes the development and sharing of novel tools that can assist in the detection of erroneous, sloppy, nonsensical, and fraudulent data. The report presented here, “GRIM-U: A GRIM-Like Observation to Establish Impossible p Values from Ranked Tests,” presents one such tool. 

We begin from the observation that p values from rank-based tests are intrinsically granular, making them amenable to forensic scrutiny. From there, we build on the principle behind the Granularity-Related Inconsistency of Means (GRIM) test – that scientific data in whole numbers only has a restricted range of means and standard deviations – to introduce the Granularity-Related Inconsistency of Mann-Whitney U (GRIM-U) test. This applies the GRIM test principle to the U values produced by the rank orders of non-parametric statistical tests, allowing us to investigate a new class of tests for their accuracy. 

To allow others to use the tool, our report provides analytic formulas, a simple Excel-spreadsheet-based calculator, a lightweight R implementation (U-Bend) for investigating the nature of the test, and heuristics for reviewers and editors to flag implausible p values efficiently. We also demonstrate the use of the tool on four opportunistically chosen published medical studies, showing how GRIM-U can be used to improve and assess the reliability of statistical reporting in medical research.